How to do peer review

Should I do it?#

Reviewing papers has many benefits. We can learn about new research; the process of reviewing forces us to critically examine a paper, which is what we should do for our own papers, and to explicitly write down why the paper should be or should not be published; we can also learn about the review process itself, which can help addressing reviewer’s comments for our own papers.

Of course, the primary drawback of paper reviewing is time commitment. Reviewing a paper can take several hours or days. Most scientific peer review is unpaid work. When declining, it is nice to provide a list of alternative reviewers, which can be quite helpful for the editor.

Mindset#

It may be helpful to frame your job not as a judge who decides the fate of the paper (although you are), but as an anonymous sage who helps the authors to make the paper better.

It has been shown that younger researchers tend to be harsher as a reviewer. If you are new, try to err on the side of giving more benefit of doubt. Papers can change a lot during the review process and authors may have really good counterarguments against your concerns.

Although you, as a reviewer, may have an outsized power in the peer review process, remember that all of your arguments should be well-supported and well-reasoned. Be especially careful not to make disparaging or personal comments. You don’t need to, and you should not, say “the authors clearly do not know much about this field”, you can instead just list the important papers that the manuscript did not cite and explain what important points have been misinterpreted by the manuscript. Just criticize the manuscript without making it personal about the authors.

Reflect on your biases. Are you commenting on the writing quality because the authors have non-English names? Are you examining the paper more critically because the authors are not in the top institutions or some developing countries?

Don’t be an a-hole. Being anonymous (but not to the editors at least) does not mean that you can say anything! The most important thing to do is thinking in the authors’ shoes. Imagine what you would feel if you receive your review report. Would you be grateful or devastated?

How To#

General rules are: don’t be obnoxious. Acknowledge good stuff and be constructive. Be sure to first check what is the expected format for the review report by the journal/conference.

Summary#

Usually it is a good idea to start the review with a summary of the manuscript. Try to write this summary, as you actively read the paper, in your own words. I’d argue that this active process of writing a summary helps you process the paper better. You can then summarize the primary contributions and the relevance and importance of the paper (note that journals like PLOS ONE explicitly ask not to evaluate the potential impact).

Recommendation#

Ask yourself whether the conclusion of the paper is supported well enough by the evidence presented by the manuscript. This leads to your overall recommendation: whether the paper can be accepted in principle or the flaws are critical enough that the paper should be rejected.

“Can be accepted in principle” means that the core idea and premise are sound—that with better execution, more evidence, or clearer writing, the paper could become publishable in the given venue. Think of it like shopping for a house and looking at the “bones”: some houses need cosmetic work but have great structure, while others have fundamental problems that no amount of renovation can fix. Similarly, some studies have a weak premise that cannot be salvaged without essentially becoming a different paper.

If the paper can be accepted in principle but still requires a lot of work—substantial new analyses, major restructuring, or significant additional evidence—that would be a major revision. If the bones are solid and only moderate changes are needed, that’s a minor revision. If the premise itself is flawed, or the paper is fundamentally not a good fit for the venue, then reject—but explain clearly why, so the authors can either rethink the study or find a more appropriate venue.

Detailed comments#

The following paragraphs should support your decision. Try your best to be constructive. Starting from the biggest concerns or comments, lay out the issues you found. Explain each issue well with enough substantive evidence. “This method is not good” or “The paper does not cite many relevant papers” are not enough. Explain why the method is not good; provide specific citations that the paper missed. Ideally, provide potential ways to address your concerns.

Resources#

• How to Write a Review by Jess Calarco
• Refereeing a journal article by Rob Hyndman
  • Referee reports
  • Part 1: Reading
  • Part 2: Making a recommendation
  • Part 3: Writing the feedback
• How to peer-review by Matt Might
• How to review a paper
• Introduction to refereeing (IOP)